HyperLoom possibilities for executing scientific workflows on the cloud

We have developed HyperLoom - a platform for defining and executing scientific workflows in large-scale HPC systems. The computational tasks in such workflows often have non-trivial dependency patterns, unknown execution time and unknown sizes of generated outputs. HyperLoom enables to efficiently execute the workflows respecting task requirements and cluster resources agnostically to the shape or size of the workflow. Although HPC infrastructures provide an unbeatable performance, they may be unavailable or too expensive especially for small to medium workloads. Moreover, for some workloads, due to HPCs not very flexible resource allocation policy, the system energy efficiency may not be optimal at some stages of the execution. In contrast, current public cloud providers such as Amazon, Google or Exoscale allow users a comfortable and elastic way of deploying, scaling and disposing a virtualized cluster of almost any size. In this paper, we describe HyperLoom virtualization and evaluate its performance in a virtualized environment using workflows of various shapes and sizes. Finally, we discuss the Hyperloom potential for its expansion to cloud environments.61140639

Real time traffic simulator for self-adaptive navigation system validation

28327

Optimization of rules selection for robot soccer strategies

Mobile embedded systems belong among the typical applications of distributed systems control in realtime. An example of a mobile control system is a robotic system. The proposal and realization of such a distributed control system represents a demanding and complex task for real-time control. In the process of robot soccer game applications, extensive data is accumulated. The reduction of such data is a possible win in a game strategy. The main topic of this article is a description of an efficient method for rule selection from a strategy. The proposed algorithm is based on the geometric representation of rules. A described problem and a proposed solution can be applied to other areas dealing with effective searching of rules in structures that also represent coordinates of the real world. Because this construed strategy describes a real space and the stores physical coordinates of real objects, our method can be used in strategic planning in the real world where we know the geographical positions of objects.Web of Science11art. no. 1

Fast algorithm for contactless partial discharge detection on remote gateway device

Detection of the high impedance fault caused by vegetation is one of the biggest challenges brought by the usage of covered conductors in medium voltage overhead power lines. One of the accompanying events of long-term contact of vegetation with the XLPE insulation is partial discharge which damages the insulation. Current systems for the detection of partial discharge have two major problems: the price and difficult installation. In this paper, an approach for the detection of the partial discharge from the data collected by the antenna is described. This approach is focused on the small computational demand and low false positive rate. Thanks to the small computational requirements, it can be run on the remote gateway devices which are collecting the data from the antenna. It is composed of four steps: outlier detection, outlier clustering, feature extraction, and classification. It is shown that this approach greatly improves the detection rate and lowers false positives compared to the previous algorithm used for partial discharge detection based on the data from an antenna, making it fit to use in the production environment.Web of Science3732130212

HyperLoom: A platform for defining and executing scientific pipelines in distributed environments

Real-world scientific applications often encompass end-to-end data processing pipelines composed of a large number of interconnected computational tasks of various granularity. We introduce HyperLoom, an open source platform for defining and executing such pipelines in distributed environments and providing a Python interface for defining tasks. HyperLoom is a self-contained system that does not use an external scheduler for the actual execution of the task. We have successfully employed HyperLoom for executing chemogenomics pipelines used in pharmaceutic industry for novel drug discovery.6

Chapter 2 The LEXIS Platform for Distributed Workflow Execution and Data Management

Artificial Intelligence, Deep Learning, Machine Learning, Supercomputin

Neural network-based urban change monitoring with deep-temporal multispectral and SAR remote sensing data

Remote-sensing-driven urban change detection has been studied in many ways for decades for a wide field of applications, such as understanding socio-economic impacts, identifying new settlements, or analyzing trends of urban sprawl. Such kinds of analyses are usually carried out manually by selecting high-quality samples that binds them to small-scale scenarios, either temporarily limited or with low spatial or temporal resolution. We propose a fully automated method that uses a large amount of available remote sensing observations for a selected period without the need to manually select samples. This enables continuous urban monitoring in a fully automated process. Furthermore, we combine multispectral optical and synthetic aperture radar (SAR) data from two eras as two mission pairs with synthetic labeling to train a neural network for detecting urban changes and activities. As pairs, we consider European Remote Sensing (ERS-1/2) and Landsat 5 Thematic Mapper (TM) for 1991-2011 and Sentinel 1 and 2 for 2017-2021. For every era, we use three different urban sites-Limassol, Rotterdam, and Liege-with at least 500 km(2) each, and deep observation time series with hundreds and up to over a thousand of samples. These sites were selected to represent different challenges in training a common neural network due to atmospheric effects, different geographies, and observation coverage. We train one model for each of the two eras using synthetic but noisy labels, which are created automatically by combining state-of-the-art methods, without the availability of existing ground truth data. To combine the benefit of both remote sensing types, the network models are ensembles of optical- and SAR-specialized sub-networks. We study the sensitivity of urban and impervious changes and the contribution of optical and SAR data to the overall solution. Our implementation and trained models are available publicly to enable others to utilize fully automated continuous urban monitoring.Web of Science1315art. no. 300

HPC For DRM - Operational Flood Management In Urban Environment

A regional flood warning system based on a combination of data processing, modelling and communication tools is proposed. The system is founded on a common framework MIKE CUSTOMISED, giving great flexibility in tailoring solutions as needed. Where more conventional flood warning systems focus mainly on discharge predictions in the main rivers the proposed system considers the whole catchment area – flood plain, as well as tributaries. Local floods on smaller streams and tributaries may cause high damages, particular in urban areas. Such cases require timely reasonably accurate forecasts for proper decision making. The MIKE SHE modelling system is used to ensure simulation of flood danger map. It also provides runoff hydrographs for the detailed hydrodynamic models of the river channel network and flood plains based on 1D / 2D approximations (MIKE FLOOD). Generated flood maps are post processed and ported to the required forms and delivered via communication channels to users. Dissemination of results is done through web pages automatically maintained by the system. Whole forecast simulation should be run in frequency of tens of minutes, which is computational power and transmission capacity demanding. Adaptation of whole system to High Performance Computer solutions (HPC) is on-going issue. This allows also parallel variant computation and real-time probabilistic assessment. IT4Innovations National Supercomputing Centre is a research institute at the VŠB – Technical University of Ostrava. This centre provides perfect platform for applications of HPC for Disaster Risk Management (DRM) and its new dynamic development for real life applications, utilised for life protection and damage minimizing. The paper contributes both to the theory of application of HPC for standard hydrodynamic modelling but also to a real life application. A pilot operational Flood Risk Mapping project is developed for the capitol city of the Czech Republic

Displacements Monitoring over Czechia by IT4S1 System for Automatised Interferometric Measurements Using Sentinel-1 Data

The Sentinel-1 satellite system continuously observes European countries at a relatively high revisit frequency of six days per orbital track. Given the Sentinel-1 configuration, most areas in Czechia are observed every 1–2 days by different tracks in a moderate resolution. This is attractive for various types of analyses by various research groups. The starting point for interferometric (InSAR) processing is an original data provided in a Single Look Complex (SLC) level. This work represents advantages of storing data augmented to a specifically corrected level of data, SLC-C. The presented database contains Czech nationwide Sentinel-1 data stored in burst units that have been pre-processed to the state of a consistent well-coregistered dataset of SLC-C. These are resampled SLC data with their phase values reduced by a topographic phase signature, ready for fast interferometric analyses (an interferogram is generated by a complex conjugate between two stored SLC-C files). The data can be used directly into multitemporal interferometry techniques, e.g., Persistent Scatterers (PS) or Small Baseline (SB) techniques applied here. A further development of the nationwide system utilising SLC-C data would lead into a dynamic state where every new pre-processed burst triggers a processing update to detect unexpected changes from InSAR time series and therefore provides a signal for early warning against a potential dangerous displacement, e.g., a landslide, instability of an engineering structure or a formation of a sinkhole. An update of the processing chain would also allow use of cross-polarised Sentinel-1 data, needed for polarimetric analyses. The current system is running at a national supercomputing centre IT4Innovations in interconnection to the Czech Copernicus Collaborative Ground Segment (CESNET), providing fast on-demand InSAR results over Czech territories. A full nationwide PS processing using data over Czechia was performed in 2017, discovering several areas of land deformation. Its downsampled version and basic findings are demonstrated within the article

Chapter 4 Data System and Data Management in a Federation of HPC/ Cloud Centers

Artificial Intelligence, Deep Learning, Machine Learning, Supercomputin